Papers with shared task
The Second Multilingual Surface Realisation Shared Task (SR’19): Overview and Evaluation Results (D19-63)
Copied to clipboard
| Challenge: | EMNLP’19 Workshop on Multilingual Surface Realisation aims to stimulate the exploration of advanced neural networks for multilingual sentence generation from Universal Dependency (UD) structures. |
| Approach: | They present results from the SR'19 Shared Task, a multilingual surface realisation task organised as part of the EMNLP'19 Workshop on Multilingual Surface Realisation. |
| Outcome: | The SR'19 shared task was organised as part of the EMNLP'19 Workshop on Multilingual Surface Realisation . it consisted of two tracks with different levels of complexity . the shallow track was offered in eleven, and the deep track in three languages . |
The FEVER2.0 Shared Task (D19-66)
Copied to clipboard
| Challenge: | Existing deep neural models are becoming more complex and difficult to understand and characterize their behaviour. |
| Approach: | They present the results of the second Fact Extraction and VERification (FEVER2.0) Shared Task. |
| Outcome: | The proposed task was based on the second Fact Extraction and VERification (FEVER2.0) shared task. |
A Deep Learning-Based System for PharmaCoNER (D19-57)
Copied to clipboard
Ying Xiong, Yedan Shen, Yuanhang Huang, Shuai Chen, Buzhou Tang, Xiaolong Wang, Qingcai Chen, Jun Yan, Yi Zhou
| Challenge: | Efficient access to mentions of clinical entities is very important for using clinical text. |
| Approach: | They developed a pipeline system based on deep learning methods for this shared task . it achieves a micro-average F1-score of 0.9105 on track 1 and a mini-average LSTM score of 0.8391 on track 2 . |
| Outcome: | The proposed system achieves a micro-average F1-score of 0.9105 on track 1 and a mini-average score of 0.8391 on track 2. |
A Neural Pipeline Approach for the PharmaCoNER Shared Task using Contextual Exhaustive Models (D19-57)
Copied to clipboard
| Challenge: | NER and concept indexing perform named entity recognition and concept identifiers (CUIs) in a knowledge base. |
| Approach: | They propose a neural pipeline approach that performs named entity recognition (NER) and concept indexing (CI) they use bi-LSTM to capture the semantic information of a sequence and classify them into entities or no entities . |
| Outcome: | The proposed approach performs named entity recognition (NER) and concept indexing (CI) which links them to concept unique identifiers (CUIs) in a knowledge base. |
Jeff Da at COIN - Shared Task: BIG MOOD: Relating Transformers to Explicit Commonsense Knowledge (D19-60)
Copied to clipboard
| Challenge: | Recent studies show that large-scale pre-training models can be effective for large datasets. |
| Approach: | They propose a method of integrating contextual embeddings with commonsense graph embeddINGs by preprocessing knowledge bases and aligning tokens between misaligned tokenization methods. |
| Outcome: | The proposed method achieves higher accuracy than BERT and scores highest without pretraining. |
Pingan Smart Health and SJTU at COIN - Shared Task: utilizing Pre-trained Language Models and Common-sense Knowledge in Machine Reading Tasks (D19-60)
Copied to clipboard
| Challenge: | Existing approaches to represent knowledge in the low-dimensional space are to leverage large-scale unsupervised text corpus to train fixed or contextual representations. |
| Approach: | They propose to leverage large-scale unsupervised text corpus to train fixed or contextual language representations and to express knowledge into a knowledge graph (KG) they incorporate distributional representations of a KG onto the representations from pre-trained language models, via simply concatenation or multi-head attention. |
| Outcome: | The proposed models outperform the other models on the COIN: COmmonsense INference in Natural Language Processing (COIN) Workshop datasets. |
Fine-Tuned Neural Models for Propaganda Detection at the Sentence and Fragment levels (D19-50)
Copied to clipboard
| Challenge: | The system was evaluated on a unified development set without distributing the gold labels. |
| Approach: | They propose to use fine-grained propaganda detection to build models that can explain why an article is propagandistic. |
| Outcome: | The proposed model performed on all eighteen propaganda techniques in the corpus of the shared task. |
Chains-of-Reasoning at TextGraphs 2019 Shared Task: Reasoning over Chains of Facts for Explainable Multi-hop Inference (D19-53)
Copied to clipboard
| Challenge: | EMNLP 2019 shared task on 'Multi-hop Inference Explanation Regeneration' identifies chains of facts relevant to explain an answer to an elementary science examination question. |
| Approach: | They propose a system that identifies chains of facts relevant to explain an answer to an elementary science examination question. |
| Outcome: | The proposed system outperforms the second best system by 14.95 points on the mean average precision (MAP) metric. |
Understanding BERT performance in propaganda analysis (D19-50)
Copied to clipboard
| Challenge: | Despite the challenging nature of the shared task, our pretrained BERT model scored 0.62 F1 on the test set and ranked third among 25 teams who participated in the contest. |
| Approach: | They propose to use a dataset to fine-tune a model for propaganda analysis at sentence level to determine whether a text is 'propaganda' and to examine false-positive cases. |
| Outcome: | The proposed model scored 0.62 F1 on the test set and ranked third among 25 teams who participated in the shared task. |
From Research to Production and Back: Ludicrously Fast Neural Machine Translation (D19-56)
Copied to clipboard
Young Jin Kim, Marcin Junczys-Dowmunt, Hany Hassan, Alham Fikri Aji, Kenneth Heafield, Roman Grundkiewicz, Nikolay Bogoychev
| Challenge: | Using the dominating submissions to the previous edition of the shared task, we develop improved teacher-student training via multi-agent dual-learning and noisy backward-forward translation for Transformer-based student models. |
| Approach: | They propose to use multi-agent dual-learning and noisy backward-forward translation to improve teacher-student training for Transformer-based student models. |
| Outcome: | The proposed model outperforms submissions to the previous edition of the WNGT efficiency shared task by 4 BLEU points and 10 BLUE points respectively. |
Multi-Agent Orchestration for Terminology-Constrained Machine Translation in Industrial Localization (2026.acl-industry)
Copied to clipboard
| Challenge: | Accurate terminology is a non-negotiable requirement in industrial localization processes. |
| Approach: | They propose a multi-agent LLM pipeline that orchestrates four specialized agents for terminology-constrained machine translation. |
| Outcome: | The proposed system achieves 99.4% average accuracy while outperforming other systems on the WMT25 Terminology Translation benchmark. |
Polyglot Semantic Role Labeling (P18-2)
Copied to clipboard
| Challenge: | Existing approaches to multilingual semantic dependency parsing treat languages independently, without exploiting similarities between semantic structures across languages. |
| Approach: | They propose to combine resources from different languages in a CoNLL 2009 shared task to build a single polyglot semantic dependency parser. |
| Outcome: | The proposed model outperforms monolingual training on a CoNLL 2009 dataset with training data from multiple languages and representations using multilingual word vectors. |
Unsupervised Parallel Sentence Extraction with Parallel Segment Detection Helps Machine Translation (P19-1)
Copied to clipboard
| Challenge: | Recent advances in unsupervised bilingual word embeddings make it possible to mine parallel sentences from comparable corpora. |
| Approach: | They propose a strong unsupervised system for parallel sentence mining based on cosine similarities of source and target words . they show that parallel sentences mined from real-life sources improve unsupervised MT . |
| Outcome: | The proposed system improves unsupervised MT on three language pairs. |
Non-Autoregressive Machine Translation: It’s Not as Fast as it Seems (2022.naacl-main)
Copied to clipboard
| Challenge: | Efficient machine translation models are commercially important as they can increase inference speeds, reduce costs and carbon emissions. |
| Approach: | They compare NAR models with autoregressive models to evaluate their performance . they point out flaws in evaluation methodology and argue for consistent evaluation . |
| Outcome: | The proposed model is faster on GPUs, but slower under more realistic usage conditions. |
A Tale of Three Parsers: Towards Diagnostic Evaluation for Meaning Representation Parsing (2020.lrec-1)
Copied to clipboard
| Challenge: | Empirical results suggest that the proposed methodology can be meaningfully applied to parsing into graph-structured target representations, uncovering hitherto unknown properties of the different approaches. |
| Approach: | They propose to map from natural language utterances to graph-based encodings of its semantic structure using contrastive and diagnostic evaluation techniques. |
| Outcome: | The proposed method can be meaningfully applied to parsing into graph-structured target representations, uncovering hitherto unknown properties of the different systems that can inform future development and cross-fertilization across approaches. |
Extraction of the Argument Structure of Tokyo Metropolitan Assembly Minutes: Segmentation of Question-and-Answer Sets (2020.lrec-1)
Copied to clipboard
Keiichi Takamaru, Yasutomo Kimura, Hideyuki Shibuki, Hokuto Ototake, Yuzu Uchida, Kotaro Sakamoto, Madoka Ishioroshi, Teruko Mitamura, Noriko Kando
| Challenge: | a study analyzed local assembly minutes in Japan using a unified format . local assembly minute data is expensive to analyze because of the different ways they are released to the public. |
| Approach: | They construct a corpus of Japanese local assembly minutes based on local autonomy law . they structured all statements in assembly minutes and extracted question and answer pairs . |
| Outcome: | The results show that the minutes are the primary information for local politics. |
Evaluating Diversity of Multiword Expressions in Annotated Text (2022.coling-1)
Copied to clipboard
| Challenge: | Using the extensive formalization and measures of diversity developed in ecology, we evaluate the variety and balance of multiword expression annotation produced by automatic annotation systems. |
| Approach: | They propose to use the formalization and measures of diversity developed in ecology to evaluate the variety and balance of multiword expression annotation produced by automatic annotation systems. |
| Outcome: | The proposed measures validate or invalidate their pertinence for multiword expressions in annotated texts. |
Miss Tools and Mr Fruit: Emergent Communication in Agents Learning about Object Affordances (P19-1)
Copied to clipboard
| Challenge: | Recent research studies communication emergence in communities of deep network agents assigned a joint task . authors propose a game meeting many desiderata for a natural communication environment . |
| Approach: | They propose a task capturing aspects of the human environment and human conversation . they propose 'game' meeting many desiderata for a natural communication environment . |
| Outcome: | The proposed task captures aspects of human environment and human conversation, but the agents develop multiple idiolects, resulting in a common language. |
Co-Teaching Student-Model through Submission Results of Shared Task (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Shared tasks require participants to submit only system outputs and descriptions. |
| Approach: | They propose to utilize all system outputs in a shared task to build a unified system that performs better than the task's single best system. |
| Outcome: | The proposed scheme outperforms the best system in the SHINRA2019-JP shared task with nine participants. |
A Multilingual Dataset for Evaluating Parallel Sentence Extraction from Comparable Corpora (L18-1)
Copied to clipboard
| Challenge: | BUCC Shared Task aims to extract parallel sentences from comparable corporad . resulting corpus contains about 3.5 million distinct sentences in english, french, german, Russian, and Chinese . |
| Approach: | They present challenges faced to build a parallel sentences dataset from comparable corporad . they emphasize issues faced to include Chinese as one of the languages . |
| Outcome: | The 2017 BUCC Shared Task was a first for this task . the dataset contains 3.5 million sentences in English, French, German, Russian, and Chinese . |
Reproduction and Revival of the Argument Reasoning Comprehension Task (2020.lrec-1)
Copied to clipboard
| Challenge: | Reproduction of scientific results is essential for scientific development across all disciplines. |
| Approach: | They evaluate scientific reproduction of arguments reasoning comprehension systems . they find reproducing results of previous work is a basic requirement for validating hypothesis . |
| Outcome: | The proposed systems were compared with the revised data set and scored in line with the results of the argument reasoning comprehension task. |